1. Field of the Invention
The present invention relates to a method for generating free viewpoint video image in three-dimensional movement and a recording medium thereof, more particularly to a method in which a plurality of video cameras (hereinafter referred to as cameras) are disposed on an identical horizontal plane such that they surround an object while another camera is disposed at the vertex so as to take pictures of the object with these cameras and generate free viewpoint images in three-dimensional movement and a recording medium for recording a program for the same method.
2. Description of the Related Art
Conventionally, there has been proposed a method in which to generate and display video image at an arbitrary viewpoint position using a group of video images taken at a plurality of the viewpoint positions, data of the actually taken video images is read into a memory and video image at the arbitrary viewpoint position is generated by calculating the brightness value of each pixel, based on a concept of ray space, that each video image data is a set of rays flying in three-dimensional space.
Here, the concept of ray space will be explained. In the three-dimensional space, rays are emitted by a light source and reflected light of an object. A ray traversing a certain position within three-dimensional space is uniquely determined by six variables indicating that position (X, Y, Z), direction (θ, φ) and time t. If a function for indicating the light intensity of this ray is defined as F by paying attention to a certain time t=0 for simplification, data of a ray group in three-dimensional space is expressed as F (X, Y, Z, θ, φ) and the group of rays within the three-dimensional space is described as five-dimensional parameter space. This parameter space and its partial space are called ray space.
First, a case of using a plane recording ray space will be described with reference to FIG. 7.
Now, attention is paid to a group of rays passing a plane called Z=Zc. This plane is called a reference plane and a two-dimensional coordinate system (P, Q) is defined on this reference plane. Here, in this two-dimensional coordinate system (P, Q), with an intersection point thereof with the Z-axis as an origin, the P-axis is set in parallel to the X-axis and the Q-axis is set in parallel to the Y-axis. If a horizontal plane (X-Z plane) perpendicular to the Y-axis is considered and it is assumed that an azimuth difference in the vertical direction is not considered (Y=0, φ=0), a real zone is as shown in FIG. 7. A group of rays emitted from the reference plane is described as F (P, θ) by a position P and an angle θ. Therefore, for a group of rays passing a point (Xc, Zc) in the real zone, the following relationship is established.P=Xc−Zc tan θ  (1)
Here, if a variable of “u=tan θ” is defined, the following equation is converted as follows.P=Xc−uZc   (2)
Therefore, on the plane recording ray space, a single ray within the real zone is mapped as a point and its light intensity, namely, color information is recorded there. Further, from the equation (2), it is apparent that a ray group passing a certain point within the real zone is mapped to a straight line on P-u space.
FIG. 8 shows a state in which ray observed at a viewpoint position (Xc, Zc) within the real zone has been mapped to the P-u space. In the meantime, the P-u space constitutes the partial space of the above-described five-dimensional ray space. The above matters also have been described in Japanese Patent Application Laid-Open Nos. 10-111951 and 2004-258775.
Next, a case of using a cylindrical recording ray space will be described with reference to FIG. 9. Meanwhile, the cylindrical recording ray space has been disclosed in Japanese Patent Application Laid-Open No. 2008-15756, which is a patent application by this assignee.
Now, attention is paid to a ray group propagated at an azimuth of θ=θc at a certain time t=0. A plane which passes the Y-axis and is perpendicular to the direction of the propagation of this ray group is called a reference plane and a two-dimensional coordinate system (P, Q) is defined on this reference plane. Here, in this two-dimensional coordinate system (P, Q), with an origin of the world coordinate system as an origin position thereof, the Q axis is set in parallel to the Y-axis and the P-axis is set in parallel to both the direction of the propagation of the ray group and the Q-axis. If a horizontal plane (X-Z plane) perpendicular to the Y-axis is considered and it is assumed that no azimuth difference in the vertical direction is taken into account (Y=0, φ=0), the real zone is as shown in FIG. 9. A ray group propagated in a direction perpendicular to the reference plane is described as F (P, θ) with two variables, position P and angle θ. Therefore, the following relationship is established for the ray group passing a certain point (Xc, Zc) within the real zone.P=Xc cos θ−Zc sin θ  (3)
Therefore, on the cylindrical recording ray space, a single ray within the real zone is mapped as a point and the light intensity, namely, color information is recorded there. From the equation (3), it is apparent that the ray group passing a certain point within the real zone is mapped to a sine curve on the P-θ space.
FIG. 10 shows a state in which ray observed at a viewpoint position (Xc, Zc) within the real zone has been mapped to the P-θ space. In the meantime, the P-θ space constitutes the partial space of the above-described five-dimensional ray space.
To reconstruct an image at an arbitrary viewpoint position from this ray space at a high precision, originally, a dimension along the Q-axis direction, that is, the dimension in the vertical direction is necessary. However, in that case, the ray space data needs to form at least a four-dimensional space of P-Q-θ-φ, so that the ray space data possesses a very large data quantity. Thus, until now, only the P-θ space (P-u space), which is a partial space of the ray space, has been considered. Further, it is considered very redundant to make the entire coordinates of the ray space possess color information. The reason is that even if only the P-θ space (P-u space) is considered, pixel information in the Q-axis direction is necessary for reconstructing images and therefore the ray space turns to three-dimensions, where the light intensity of each ray needs to be recorded. Then, there is a method in which ray space arithmetic operation is performed for all pixels of the image to be reconstructed so as to obtain the brightness value from a multi-viewpoint image (image taken from a plurality of different viewpoint positions) read into a memory. In the meantime, the ray space arithmetic operation refers to an arithmetic operation to be performed based on the equation (2) and the equation (3) in the P-θ space (P-u space).
According to the above-described conventional example, to generate and display an image at an arbitrary viewpoint position at a real time corresponding to the movement of an operator, high-speed ray space arithmetic operation is needed. To perform the arithmetic operation, an operation of reading pixel data by accessing a multi-viewpoint image at random must be done. That is, a high-speed random access to the multi-viewpoint image is required. Then, the above-mentioned example has adopted a way of reading the P-θ space (P-u space) and the multi-viewpoint image into the memory before the arithmetic operation.
However, an image from a viewpoint which looks down on an object from above could not be synthesized because according to the conventional method, information about azimuth difference in the vertical direction was not obtained.